{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yfv52r2G33jY"
   },
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/AI4Finance-Foundation/FinRL-Tutorials/blob/master/1-Introduction/Stock_NeurIPS2018_ElegantRL.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gXaoZs2lh1hi"
   },
   "source": [
    "# Deep Reinforcement Learning for Stock Trading from Scratch: Multiple Stock Trading\n",
    "\n",
    "* **Pytorch Version** \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lGunVt8oLCVS"
   },
   "source": [
    "# Content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HOzAKQ-SLGX6"
   },
   "source": [
    "* [1. Task Description](#0)\n",
    "* [2. Install Python packages](#1)\n",
    "    * [2.1. Install Packages](#1.1)    \n",
    "    * [2.2. A List of Python Packages](#1.2)\n",
    "    * [2.3. Import Packages](#1.3)\n",
    "    * [2.4. Create Folders](#1.4)\n",
    "* [3. Download and Preprocess Data](#2)\n",
    "* [4. Preprocess Data](#3)        \n",
    "    * [4.1. Technical Indicators](#3.1)\n",
    "    * [4.2. Perform Feature Engineering](#3.2)\n",
    "* [5. Build Market Environment in OpenAI Gym-style](#4)  \n",
    "    * [5.1. Data Split](#4.1)  \n",
    "    * [5.3. Environment for Training](#4.2)    \n",
    "* [6. Train DRL Agents](#5)\n",
    "* [7. Backtesting Performance](#6)  \n",
    "    * [7.1. BackTestStats](#6.1)\n",
    "    * [7.2. BackTestPlot](#6.2)   \n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sApkDlD9LIZv"
   },
   "source": [
    "<a id='0'></a>\n",
    "# Part 1. Task Discription"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HjLD2TZSLKZ-"
   },
   "source": [
    "We train a DRL agent for stock trading. This task is modeled as a Markov Decision Process (MDP), and the objective function is maximizing (expected) cumulative return.\n",
    "\n",
    "We specify the state-action-reward as follows:\n",
    "\n",
    "* **State s**: The state space represents an agent's perception of the market environment. Just like a human trader analyzing various information, here our agent passively observes many features and learns by interacting with the market environment (usually by replaying historical data).\n",
    "\n",
    "* **Action a**: The action space includes allowed actions that an agent can take at each state. For example, a ∈ {−1, 0, 1}, where −1, 0, 1 represent\n",
    "selling, holding, and buying. When an action operates multiple shares, a ∈{−k, ..., −1, 0, 1, ..., k}, e.g.. \"Buy\n",
    "10 shares of AAPL\" or \"Sell 10 shares of AAPL\" are 10 or −10, respectively\n",
    "\n",
    "* **Reward function r(s, a, s′)**: Reward is an incentive for an agent to learn a better policy. For example, it can be the change of the portfolio value when taking a at state s and arriving at new state s',  i.e., r(s, a, s′) = v′ − v, where v′ and v represent the portfolio values at state s′ and s, respectively\n",
    "\n",
    "\n",
    "**Market environment**: 30 consituent stocks of Dow Jones Industrial Average (DJIA) index. Accessed at the starting date of the testing period.\n",
    "\n",
    "\n",
    "The data for this case study is obtained from Yahoo Finance API. The data contains Open-High-Low-Close price and volume.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ffsre789LY08"
   },
   "source": [
    "<a id='1'></a>\n",
    "# Part 2. Install Python Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Uy5_PTmOh1hj"
   },
   "source": [
    "<a id='1.1'></a>\n",
    "## 2.1. Install packages\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "mPT0ipYE28wL",
    "outputId": "0419404b-38f2-4236-9cf9-3e9ea03fcb08"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
      "Requirement already satisfied: finrl==0.3.5 in /usr/local/lib/python3.7/dist-packages (0.3.5)\n",
      "Collecting elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl\n",
      "  Cloning https://github.com/AI4Finance-Foundation/ElegantRL.git to /tmp/pip-install-sgmoexxz/elegantrl_630fc058d04f4bdfbcc62901201fb635\n",
      "  Running command git clone -q https://github.com/AI4Finance-Foundation/ElegantRL.git /tmp/pip-install-sgmoexxz/elegantrl_630fc058d04f4bdfbcc62901201fb635\n",
      "Collecting pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2\n",
      "  Cloning https://github.com/quantopian/pyfolio.git to /tmp/pip-install-sgmoexxz/pyfolio_5be79995f87e4711b154852ba23f3708\n",
      "  Running command git clone -q https://github.com/quantopian/pyfolio.git /tmp/pip-install-sgmoexxz/pyfolio_5be79995f87e4711b154852ba23f3708\n",
      "Requirement already satisfied: ray[tune] in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (2.0.0)\n",
      "Requirement already satisfied: pandas>=1.1.5 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (1.3.5)\n",
      "Requirement already satisfied: gym>=0.17 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (0.21.0)\n",
      "Requirement already satisfied: jqdatasdk in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (1.8.10)\n",
      "Requirement already satisfied: pre-commit in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (2.20.0)\n",
      "Requirement already satisfied: wrds in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (3.1.2)\n",
      "Requirement already satisfied: yfinance in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (0.1.74)\n",
      "Requirement already satisfied: elegantrl in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (0.3.2)\n",
      "Requirement already satisfied: wheel>=0.33.6 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (0.37.1)\n",
      "Requirement already satisfied: exchange-calendars==3.6.3 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (3.6.3)\n",
      "Requirement already satisfied: scikit-learn>=0.21.0 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (1.0.2)\n",
      "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (3.2.2)\n",
      "Requirement already satisfied: ccxt==1.66.32 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (1.66.32)\n",
      "Requirement already satisfied: pytest in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (3.6.4)\n",
      "Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (1.21.6)\n",
      "Requirement already satisfied: gputil in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (1.4.0)\n",
      "Requirement already satisfied: setuptools==59.5.0 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (59.5.0)\n",
      "Requirement already satisfied: stockstats>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (0.4.1)\n",
      "Requirement already satisfied: stable-baselines3[extra] in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (1.6.0)\n",
      "Requirement already satisfied: lz4 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (4.0.2)\n",
      "Requirement already satisfied: tensorboardX in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (2.5.1)\n",
      "Requirement already satisfied: alpaca-trade-api>=2.1.0 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.5) (2.3.0)\n",
      "Requirement already satisfied: pybullet in /usr/local/lib/python3.7/dist-packages (from elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.5) (3.2.5)\n",
      "Requirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (from elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.5) (1.12.1+cu113)\n",
      "Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.5) (4.6.0.66)\n",
      "Requirement already satisfied: box2d-py in /usr/local/lib/python3.7/dist-packages (from elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.5) (2.3.8)\n",
      "Requirement already satisfied: ipython>=3.2.3 in /usr/local/lib/python3.7/dist-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (7.9.0)\n",
      "Requirement already satisfied: pytz>=2014.10 in /usr/local/lib/python3.7/dist-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (2022.2.1)\n",
      "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (1.7.3)\n",
      "Requirement already satisfied: seaborn>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.11.2)\n",
      "Requirement already satisfied: empyrical>=0.5.0 in /usr/local/lib/python3.7/dist-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.5.5)\n",
      "Requirement already satisfied: requests>=2.18.4 in /usr/local/lib/python3.7/dist-packages (from ccxt==1.66.32->finrl==0.3.5) (2.28.1)\n",
      "Requirement already satisfied: cryptography>=2.6.1 in /usr/local/lib/python3.7/dist-packages (from ccxt==1.66.32->finrl==0.3.5) (37.0.4)\n",
      "Requirement already satisfied: certifi>=2018.1.18 in /usr/local/lib/python3.7/dist-packages (from ccxt==1.66.32->finrl==0.3.5) (2022.6.15)\n",
      "Requirement already satisfied: aiohttp>=3.8 in /usr/local/lib/python3.7/dist-packages (from ccxt==1.66.32->finrl==0.3.5) (3.8.1)\n",
      "Requirement already satisfied: aiodns>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from ccxt==1.66.32->finrl==0.3.5) (3.0.0)\n",
      "Requirement already satisfied: yarl==1.7.2 in /usr/local/lib/python3.7/dist-packages (from ccxt==1.66.32->finrl==0.3.5) (1.7.2)\n",
      "Requirement already satisfied: korean-lunar-calendar in /usr/local/lib/python3.7/dist-packages (from exchange-calendars==3.6.3->finrl==0.3.5) (0.2.1)\n",
      "Requirement already satisfied: toolz in /usr/local/lib/python3.7/dist-packages (from exchange-calendars==3.6.3->finrl==0.3.5) (0.12.0)\n",
      "Requirement already satisfied: pyluach in /usr/local/lib/python3.7/dist-packages (from exchange-calendars==3.6.3->finrl==0.3.5) (2.0.1)\n",
      "Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/dist-packages (from exchange-calendars==3.6.3->finrl==0.3.5) (2.8.2)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from yarl==1.7.2->ccxt==1.66.32->finrl==0.3.5) (4.1.1)\n",
      "Requirement already satisfied: multidict>=4.0 in /usr/local/lib/python3.7/dist-packages (from yarl==1.7.2->ccxt==1.66.32->finrl==0.3.5) (6.0.2)\n",
      "Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.7/dist-packages (from yarl==1.7.2->ccxt==1.66.32->finrl==0.3.5) (2.10)\n",
      "Requirement already satisfied: pycares>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from aiodns>=1.1.1->ccxt==1.66.32->finrl==0.3.5) (4.2.2)\n",
      "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp>=3.8->ccxt==1.66.32->finrl==0.3.5) (2.1.1)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.7/dist-packages (from aiohttp>=3.8->ccxt==1.66.32->finrl==0.3.5) (1.2.0)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp>=3.8->ccxt==1.66.32->finrl==0.3.5) (22.1.0)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.7/dist-packages (from aiohttp>=3.8->ccxt==1.66.32->finrl==0.3.5) (4.0.2)\n",
      "Requirement already satisfied: asynctest==0.13.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp>=3.8->ccxt==1.66.32->finrl==0.3.5) (0.13.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from aiohttp>=3.8->ccxt==1.66.32->finrl==0.3.5) (1.3.1)\n",
      "Requirement already satisfied: websocket-client<2,>=0.56.0 in /usr/local/lib/python3.7/dist-packages (from alpaca-trade-api>=2.1.0->finrl==0.3.5) (1.4.0)\n",
      "Requirement already satisfied: websockets<11,>=9.0 in /usr/local/lib/python3.7/dist-packages (from alpaca-trade-api>=2.1.0->finrl==0.3.5) (10.3)\n",
      "Requirement already satisfied: urllib3<2,>1.24 in /usr/local/lib/python3.7/dist-packages (from alpaca-trade-api>=2.1.0->finrl==0.3.5) (1.24.3)\n",
      "Requirement already satisfied: msgpack==1.0.3 in /usr/local/lib/python3.7/dist-packages (from alpaca-trade-api>=2.1.0->finrl==0.3.5) (1.0.3)\n",
      "Requirement already satisfied: PyYAML==6.0 in /usr/local/lib/python3.7/dist-packages (from alpaca-trade-api>=2.1.0->finrl==0.3.5) (6.0)\n",
      "Requirement already satisfied: deprecation==2.1.0 in /usr/local/lib/python3.7/dist-packages (from alpaca-trade-api>=2.1.0->finrl==0.3.5) (2.1.0)\n",
      "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from deprecation==2.1.0->alpaca-trade-api>=2.1.0->finrl==0.3.5) (21.3)\n",
      "Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=2.6.1->ccxt==1.66.32->finrl==0.3.5) (1.15.1)\n",
      "Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=2.6.1->ccxt==1.66.32->finrl==0.3.5) (2.21)\n",
      "Requirement already satisfied: pandas-datareader>=0.2 in /usr/local/lib/python3.7/dist-packages (from empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.9.0)\n",
      "Requirement already satisfied: importlib-metadata>=4.8.1 in /usr/local/lib/python3.7/dist-packages (from gym>=0.17->finrl==0.3.5) (4.12.0)\n",
      "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from gym>=0.17->finrl==0.3.5) (1.5.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.8.1->gym>=0.17->finrl==0.3.5) (3.8.1)\n",
      "Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (2.0.10)\n",
      "Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (5.1.1)\n",
      "Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (2.6.1)\n",
      "Requirement already satisfied: backcall in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.2.0)\n",
      "Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (4.4.2)\n",
      "Requirement already satisfied: jedi>=0.10 in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.18.1)\n",
      "Requirement already satisfied: pexpect in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (4.8.0)\n",
      "Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.7.5)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /usr/local/lib/python3.7/dist-packages (from jedi>=0.10->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.8.3)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->finrl==0.3.5) (0.11.0)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->finrl==0.3.5) (3.0.9)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->finrl==0.3.5) (1.4.4)\n",
      "Requirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from pandas-datareader>=0.2->empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (4.9.1)\n",
      "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.1.0,>=2.0.0->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (1.15.0)\n",
      "Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.1.0,>=2.0.0->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.2.5)\n",
      "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.0->finrl==0.3.5) (1.1.0)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.0->finrl==0.3.5) (3.1.0)\n",
      "Requirement already satisfied: SQLAlchemy>=1.2.8 in /usr/local/lib/python3.7/dist-packages (from jqdatasdk->finrl==0.3.5) (1.4.40)\n",
      "Requirement already satisfied: thriftpy2>=0.3.9 in /usr/local/lib/python3.7/dist-packages (from jqdatasdk->finrl==0.3.5) (0.4.14)\n",
      "Requirement already satisfied: pymysql>=0.7.6 in /usr/local/lib/python3.7/dist-packages (from jqdatasdk->finrl==0.3.5) (1.0.2)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.7/dist-packages (from SQLAlchemy>=1.2.8->jqdatasdk->finrl==0.3.5) (1.1.3)\n",
      "Requirement already satisfied: ply<4.0,>=3.4 in /usr/local/lib/python3.7/dist-packages (from thriftpy2>=0.3.9->jqdatasdk->finrl==0.3.5) (3.11)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.7/dist-packages (from pexpect->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.5) (0.7.0)\n",
      "Requirement already satisfied: cfgv>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from pre-commit->finrl==0.3.5) (3.3.1)\n",
      "Requirement already satisfied: identify>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from pre-commit->finrl==0.3.5) (2.5.3)\n",
      "Requirement already satisfied: virtualenv>=20.0.8 in /usr/local/lib/python3.7/dist-packages (from pre-commit->finrl==0.3.5) (20.16.4)\n",
      "Requirement already satisfied: toml in /usr/local/lib/python3.7/dist-packages (from pre-commit->finrl==0.3.5) (0.10.2)\n",
      "Requirement already satisfied: nodeenv>=0.11.1 in /usr/local/lib/python3.7/dist-packages (from pre-commit->finrl==0.3.5) (1.7.0)\n",
      "Requirement already satisfied: filelock<4,>=3.4.1 in /usr/local/lib/python3.7/dist-packages (from virtualenv>=20.0.8->pre-commit->finrl==0.3.5) (3.8.0)\n",
      "Requirement already satisfied: platformdirs<3,>=2.4 in /usr/local/lib/python3.7/dist-packages (from virtualenv>=20.0.8->pre-commit->finrl==0.3.5) (2.5.2)\n",
      "Requirement already satisfied: distlib<1,>=0.3.5 in /usr/local/lib/python3.7/dist-packages (from virtualenv>=20.0.8->pre-commit->finrl==0.3.5) (0.3.6)\n",
      "Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from pytest->finrl==0.3.5) (8.14.0)\n",
      "Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.7/dist-packages (from pytest->finrl==0.3.5) (1.4.1)\n",
      "Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.7/dist-packages (from pytest->finrl==0.3.5) (0.7.1)\n",
      "Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pytest->finrl==0.3.5) (1.11.0)\n",
      "Requirement already satisfied: protobuf<4.0.0,>=3.15.3 in /usr/local/lib/python3.7/dist-packages (from ray[tune]->finrl==0.3.5) (3.19.4)\n",
      "Requirement already satisfied: click<=8.0.4,>=7.0 in /usr/local/lib/python3.7/dist-packages (from ray[tune]->finrl==0.3.5) (7.1.2)\n",
      "Requirement already satisfied: jsonschema in /usr/local/lib/python3.7/dist-packages (from ray[tune]->finrl==0.3.5) (4.3.3)\n",
      "Requirement already satisfied: grpcio<=1.43.0,>=1.28.1 in /usr/local/lib/python3.7/dist-packages (from ray[tune]->finrl==0.3.5) (1.43.0)\n",
      "Requirement already satisfied: prometheus-client<0.14.0,>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from ray[tune]->finrl==0.3.5) (0.13.1)\n",
      "Requirement already satisfied: gpustat>=1.0.0b1 in /usr/local/lib/python3.7/dist-packages (from ray[tune]->finrl==0.3.5) (1.0.0rc1)\n",
      "Requirement already satisfied: smart-open in /usr/local/lib/python3.7/dist-packages (from ray[tune]->finrl==0.3.5) (5.2.1)\n",
      "Requirement already satisfied: colorful in /usr/local/lib/python3.7/dist-packages (from ray[tune]->finrl==0.3.5) (0.5.4)\n",
      "Requirement already satisfied: opencensus in /usr/local/lib/python3.7/dist-packages (from ray[tune]->finrl==0.3.5) (0.11.0)\n",
      "Requirement already satisfied: aiohttp-cors in /usr/local/lib/python3.7/dist-packages (from ray[tune]->finrl==0.3.5) (0.7.0)\n",
      "Requirement already satisfied: py-spy>=0.2.0 in /usr/local/lib/python3.7/dist-packages (from ray[tune]->finrl==0.3.5) (0.3.12)\n",
      "Requirement already satisfied: pydantic in /usr/local/lib/python3.7/dist-packages (from ray[tune]->finrl==0.3.5) (1.9.2)\n",
      "Requirement already satisfied: nvidia-ml-py<=11.495.46,>=11.450.129 in /usr/local/lib/python3.7/dist-packages (from gpustat>=1.0.0b1->ray[tune]->finrl==0.3.5) (11.495.46)\n",
      "Requirement already satisfied: psutil>=5.6.0 in /usr/local/lib/python3.7/dist-packages (from gpustat>=1.0.0b1->ray[tune]->finrl==0.3.5) (5.9.1)\n",
      "Requirement already satisfied: blessed>=1.17.1 in /usr/local/lib/python3.7/dist-packages (from gpustat>=1.0.0b1->ray[tune]->finrl==0.3.5) (1.19.1)\n",
      "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema->ray[tune]->finrl==0.3.5) (0.18.1)\n",
      "Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema->ray[tune]->finrl==0.3.5) (5.9.0)\n",
      "Requirement already satisfied: google-api-core<3.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from opencensus->ray[tune]->finrl==0.3.5) (1.31.6)\n",
      "Requirement already satisfied: opencensus-context>=0.1.3 in /usr/local/lib/python3.7/dist-packages (from opencensus->ray[tune]->finrl==0.3.5) (0.1.3)\n",
      "Requirement already satisfied: google-auth<2.0dev,>=1.25.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[tune]->finrl==0.3.5) (1.35.0)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[tune]->finrl==0.3.5) (1.56.4)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[tune]->finrl==0.3.5) (0.2.8)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[tune]->finrl==0.3.5) (4.2.4)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<2.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[tune]->finrl==0.3.5) (4.9)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[tune]->finrl==0.3.5) (0.4.8)\n",
      "Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from ray[tune]->finrl==0.3.5) (0.8.10)\n",
      "Requirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]->finrl==0.3.5) (2.8.0)\n",
      "Requirement already satisfied: ale-py==0.7.4 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]->finrl==0.3.5) (0.7.4)\n",
      "Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]->finrl==0.3.5) (7.1.2)\n",
      "Requirement already satisfied: autorom[accept-rom-license]~=0.4.2 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]->finrl==0.3.5) (0.4.2)\n",
      "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from autorom[accept-rom-license]~=0.4.2->stable-baselines3[extra]->finrl==0.3.5) (4.64.0)\n",
      "Requirement already satisfied: AutoROM.accept-rom-license in /usr/local/lib/python3.7/dist-packages (from autorom[accept-rom-license]~=0.4.2->stable-baselines3[extra]->finrl==0.3.5) (0.4.2)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.5) (0.6.1)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.5) (1.8.1)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.5) (0.4.6)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.5) (1.0.1)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.5) (3.4.1)\n",
      "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.5) (1.2.0)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.5) (1.3.1)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.5) (3.2.0)\n",
      "Requirement already satisfied: psycopg2-binary in /usr/local/lib/python3.7/dist-packages (from wrds->finrl==0.3.5) (2.9.3)\n",
      "Requirement already satisfied: mock in /usr/local/lib/python3.7/dist-packages (from wrds->finrl==0.3.5) (4.0.3)\n",
      "Requirement already satisfied: multitasking>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from yfinance->finrl==0.3.5) (0.0.11)\n",
      "Building wheels for collected packages: elegantrl\n",
      "  Building wheel for elegantrl (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for elegantrl: filename=elegantrl-0.3.3-py3-none-any.whl size=347412 sha256=d55a727feac292fdd6da401feec88b19a79ec9226e1b445efd24386280119c44\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-t611x4m2/wheels/99/85/5e/86cb3a9f47adfca5e248295e93113e1b298d60883126d62c84\n",
      "Successfully built elegantrl\n",
      "Installing collected packages: elegantrl\n",
      "  Attempting uninstall: elegantrl\n",
      "    Found existing installation: elegantrl 0.3.2\n",
      "    Uninstalling elegantrl-0.3.2:\n",
      "      Successfully uninstalled elegantrl-0.3.2\n",
      "Successfully installed elegantrl-0.3.3\n"
     ]
    },
    {
     "data": {
      "application/vnd.colab-display-data+json": {
       "pip_warning": {
        "packages": [
         "elegantrl"
        ]
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
      "Requirement already satisfied: elegantrl==0.3.3 in /usr/local/lib/python3.7/dist-packages (0.3.3)\n",
      "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from elegantrl==0.3.3) (3.2.2)\n",
      "Requirement already satisfied: box2d-py in /usr/local/lib/python3.7/dist-packages (from elegantrl==0.3.3) (2.3.8)\n",
      "Requirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (from elegantrl==0.3.3) (1.12.1+cu113)\n",
      "Requirement already satisfied: gym in /usr/local/lib/python3.7/dist-packages (from elegantrl==0.3.3) (0.21.0)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from elegantrl==0.3.3) (1.21.6)\n",
      "Requirement already satisfied: pybullet in /usr/local/lib/python3.7/dist-packages (from elegantrl==0.3.3) (3.2.5)\n",
      "Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from elegantrl==0.3.3) (4.6.0.66)\n",
      "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from gym->elegantrl==0.3.3) (1.5.0)\n",
      "Requirement already satisfied: importlib-metadata>=4.8.1 in /usr/local/lib/python3.7/dist-packages (from gym->elegantrl==0.3.3) (4.12.0)\n",
      "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.8.1->gym->elegantrl==0.3.3) (4.1.1)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.8.1->gym->elegantrl==0.3.3) (3.8.1)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->elegantrl==0.3.3) (0.11.0)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->elegantrl==0.3.3) (3.0.9)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->elegantrl==0.3.3) (2.8.2)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->elegantrl==0.3.3) (1.4.4)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib->elegantrl==0.3.3) (1.15.0)\n"
     ]
    }
   ],
   "source": [
    "## install finrl library\n",
    "!pip install wrds\n",
    "!pip install swig\n",
    "!pip install finrl==0.3.5\n",
    "\n",
    "## instal elegantrl\n",
    "!pip install elegantrl==0.3.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "osBHhVysOEzi"
   },
   "source": [
    "\n",
    "<a id='1.2'></a>\n",
    "## 2.2. A list of Python packages \n",
    "* Yahoo Finance API\n",
    "* pandas\n",
    "* numpy\n",
    "* matplotlib\n",
    "* stockstats\n",
    "* OpenAI gym\n",
    "* stable-baselines\n",
    "* tensorflow\n",
    "* pyfolio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nGv01K8Sh1hn"
   },
   "source": [
    "<a id='1.3'></a>\n",
    "## 2.3. Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lPqeTTwoh1hn"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "# matplotlib.use('Agg')\n",
    "import datetime\n",
    "\n",
    "%matplotlib inline\n",
    "from finrl.meta.preprocessor.yahoodownloader import YahooDownloader\n",
    "from finrl.meta.preprocessor.preprocessors import FeatureEngineer, data_split\n",
    "from finrl.meta.env_stock_trading.env_stocktrading import StockTradingEnv\n",
    "from finrl.agents.elegantrl.models import DRLAgent\n",
    "from stable_baselines3.common.logger import configure\n",
    "from finrl.meta.data_processor import DataProcessor\n",
    "\n",
    "from finrl.plot import backtest_stats, backtest_plot, get_daily_return, get_baseline\n",
    "from pprint import pprint\n",
    "\n",
    "# from elegantrl.agent import AgentDDPG\n",
    "# from elegantrl.agent import AgentPPO\n",
    "# from elegantrl.agent import AgentTD3\n",
    "# from elegantrl.agent import AgentSAC\n",
    "\n",
    "import sys\n",
    "sys.path.append(\"../FinRL\")\n",
    "\n",
    "import itertools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T2owTj985RW4"
   },
   "source": [
    "<a id='1.4'></a>\n",
    "## 2.4. Create Folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RtUc_ofKmpdy"
   },
   "outputs": [],
   "source": [
    "from finrl import config\n",
    "from finrl import config_tickers\n",
    "import os\n",
    "from finrl.main import check_and_make_directories\n",
    "from finrl.config import (\n",
    "    DATA_SAVE_DIR,\n",
    "    TRAINED_MODEL_DIR,\n",
    "    TENSORBOARD_LOG_DIR,\n",
    "    RESULTS_DIR,\n",
    "    INDICATORS,\n",
    "    TRAIN_START_DATE,\n",
    "    TRAIN_END_DATE,\n",
    "    TEST_START_DATE,\n",
    "    TEST_END_DATE,\n",
    "    TRADE_START_DATE,\n",
    "    TRADE_END_DATE,\n",
    ")\n",
    "check_and_make_directories([DATA_SAVE_DIR, TRAINED_MODEL_DIR, TENSORBOARD_LOG_DIR, RESULTS_DIR])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A289rQWMh1hq"
   },
   "source": [
    "<a id='2'></a>\n",
    "# Part 3. Download Data\n",
    "Yahoo Finance provides stock data, financial news, financial reports, etc. Yahoo Finance is free.\n",
    "* FinRL uses a class **YahooDownloader** in FinRL-Meta to fetch data via Yahoo Finance API\n",
    "* Call Limit: Using the Public API (without authentication), you are limited to 2,000 requests per hour per IP (or up to a total of 48,000 requests a day)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NPeQ7iS-LoMm"
   },
   "source": [
    "\n",
    "\n",
    "-----\n",
    "class YahooDownloader:\n",
    "    Retrieving daily stock data from\n",
    "    Yahoo Finance API\n",
    "\n",
    "    Attributes\n",
    "    ----------\n",
    "        start_date : str\n",
    "            start date of the data (modified from config.py)\n",
    "        end_date : str\n",
    "            end date of the data (modified from config.py)\n",
    "        ticker_list : list\n",
    "            a list of stock tickers (modified from config.py)\n",
    "\n",
    "    Methods\n",
    "    -------\n",
    "    fetch_data()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "h3XJnvrbLp-C",
    "outputId": "9a729d99-9a47-446f-819f-40af9f749577"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'2020-07-31'"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# from config.py, TRAIN_START_DATE is a string\n",
    "TRAIN_START_DATE\n",
    "# from config.py, TRAIN_END_DATE is a string\n",
    "TRAIN_END_DATE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FUnY8WEfLq3C"
   },
   "outputs": [],
   "source": [
    "TRAIN_START_DATE = '2009-01-01'\n",
    "TRAIN_END_DATE = '2020-07-01'\n",
    "TRADE_START_DATE = '2020-07-01'\n",
    "TRADE_END_DATE = '2021-10-31'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yCKm4om-s9kE",
    "outputId": "cef79352-a24d-4b6f-9c4e-d51862252e99"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "Shape of DataFrame:  (94331, 8)\n"
     ]
    }
   ],
   "source": [
    "df = YahooDownloader(start_date = TRAIN_START_DATE,\n",
    "                     end_date = TRADE_END_DATE,\n",
    "                     ticker_list = config_tickers.DOW_30_TICKER).fetch_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JzqRRTOX6aFu",
    "outputId": "04aab664-1304-46b1-8bd9-7f2dc0327cda"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['AXP', 'AMGN', 'AAPL', 'BA', 'CAT', 'CSCO', 'CVX', 'GS', 'HD', 'HON', 'IBM', 'INTC', 'JNJ', 'KO', 'JPM', 'MCD', 'MMM', 'MRK', 'MSFT', 'NKE', 'PG', 'TRV', 'UNH', 'CRM', 'VZ', 'V', 'WBA', 'WMT', 'DIS', 'DOW']\n"
     ]
    }
   ],
   "source": [
    "print(config_tickers.DOW_30_TICKER)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CV3HrZHLh1hy",
    "outputId": "deb1559e-59f1-4c20-e3b3-7f08c8b2ba10"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(94331, 8)"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "4hYkeaPiICHS",
    "outputId": "02aafee1-21fe-45ff-f237-2cfa46061b49"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-02ef4025-a2cf-481d-b621-73231ed554f9\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>tic</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>3.067143</td>\n",
       "      <td>3.251429</td>\n",
       "      <td>3.041429</td>\n",
       "      <td>2.767330</td>\n",
       "      <td>746015200</td>\n",
       "      <td>AAPL</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>58.590000</td>\n",
       "      <td>59.080002</td>\n",
       "      <td>57.750000</td>\n",
       "      <td>44.523743</td>\n",
       "      <td>6547900</td>\n",
       "      <td>AMGN</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>18.570000</td>\n",
       "      <td>19.520000</td>\n",
       "      <td>18.400000</td>\n",
       "      <td>15.477420</td>\n",
       "      <td>10955700</td>\n",
       "      <td>AXP</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>42.799999</td>\n",
       "      <td>45.560001</td>\n",
       "      <td>42.779999</td>\n",
       "      <td>33.941086</td>\n",
       "      <td>7010200</td>\n",
       "      <td>BA</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>44.910000</td>\n",
       "      <td>46.980000</td>\n",
       "      <td>44.709999</td>\n",
       "      <td>31.942247</td>\n",
       "      <td>7117200</td>\n",
       "      <td>CAT</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-02ef4025-a2cf-481d-b621-73231ed554f9')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-02ef4025-a2cf-481d-b621-73231ed554f9 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-02ef4025-a2cf-481d-b621-73231ed554f9');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "         date       open       high        low      close     volume   tic  \\\n",
       "0  2009-01-02   3.067143   3.251429   3.041429   2.767330  746015200  AAPL   \n",
       "1  2009-01-02  58.590000  59.080002  57.750000  44.523743    6547900  AMGN   \n",
       "2  2009-01-02  18.570000  19.520000  18.400000  15.477420   10955700   AXP   \n",
       "3  2009-01-02  42.799999  45.560001  42.779999  33.941086    7010200    BA   \n",
       "4  2009-01-02  44.910000  46.980000  44.709999  31.942247    7117200   CAT   \n",
       "\n",
       "   day  \n",
       "0    4  \n",
       "1    4  \n",
       "2    4  \n",
       "3    4  \n",
       "4    4  "
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(['date','tic'],ignore_index=True).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uqC6c40Zh1iH"
   },
   "source": [
    "# Part 4: Preprocess Data\n",
    "We need to check for missing data and do feature engineering to convert the data point into a state.\n",
    "* **Adding technical indicators**. In practical trading, various information needs to be taken into account, such as historical prices, current holding shares, technical indicators, etc. Here, we demonstrate two trend-following technical indicators: MACD and RSI.\n",
    "* **Adding turbulence index**. Risk-aversion reflects whether an investor prefers to protect the capital. It also influences one's trading strategy when facing different market volatility level. To control the risk in a worst-case scenario, such as financial crisis of 2007–2008, FinRL employs the turbulence index that measures extreme fluctuation of asset price."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PmKP-1ii3RLS",
    "outputId": "490cfd58-fb68-4d79-bc30-326a0d1d5bdc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully added technical indicators\n",
      "[*********************100%***********************]  1 of 1 completed\n",
      "Shape of DataFrame:  (3229, 8)\n",
      "Successfully added vix\n",
      "Successfully added turbulence index\n"
     ]
    }
   ],
   "source": [
    "fe = FeatureEngineer(\n",
    "                    use_technical_indicator=True,\n",
    "                    tech_indicator_list = INDICATORS,\n",
    "                    use_vix=True,\n",
    "                    use_turbulence=True,\n",
    "                    user_defined_feature = False)\n",
    "\n",
    "processed = fe.preprocess_data(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Kixon2tR3RLT"
   },
   "outputs": [],
   "source": [
    "list_ticker = processed[\"tic\"].unique().tolist()\n",
    "list_date = list(pd.date_range(processed['date'].min(),processed['date'].max()).astype(str))\n",
    "combination = list(itertools.product(list_date,list_ticker))\n",
    "\n",
    "processed_full = pd.DataFrame(combination,columns=[\"date\",\"tic\"]).merge(processed,on=[\"date\",\"tic\"],how=\"left\")\n",
    "processed_full = processed_full[processed_full['date'].isin(processed['date'])]\n",
    "processed_full = processed_full.sort_values(['date','tic'])\n",
    "\n",
    "processed_full = processed_full.fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 407
    },
    "id": "grvhGJJII3Xn",
    "outputId": "31022347-ebf6-4e2d-9555-c78502c51d42"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-cb68e2b7-fcdf-4531-bf90-cce0224fb057\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>tic</th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>day</th>\n",
       "      <th>macd</th>\n",
       "      <th>boll_ub</th>\n",
       "      <th>boll_lb</th>\n",
       "      <th>rsi_30</th>\n",
       "      <th>cci_30</th>\n",
       "      <th>dx_30</th>\n",
       "      <th>close_30_sma</th>\n",
       "      <th>close_60_sma</th>\n",
       "      <th>vix</th>\n",
       "      <th>turbulence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>AAPL</td>\n",
       "      <td>3.067143</td>\n",
       "      <td>3.251429</td>\n",
       "      <td>3.041429</td>\n",
       "      <td>2.767330</td>\n",
       "      <td>746015200.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.990894</td>\n",
       "      <td>2.660558</td>\n",
       "      <td>100.0</td>\n",
       "      <td>66.666667</td>\n",
       "      <td>100.0</td>\n",
       "      <td>2.767330</td>\n",
       "      <td>2.767330</td>\n",
       "      <td>39.189999</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>AMGN</td>\n",
       "      <td>58.590000</td>\n",
       "      <td>59.080002</td>\n",
       "      <td>57.750000</td>\n",
       "      <td>44.523743</td>\n",
       "      <td>6547900.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.990894</td>\n",
       "      <td>2.660558</td>\n",
       "      <td>100.0</td>\n",
       "      <td>66.666667</td>\n",
       "      <td>100.0</td>\n",
       "      <td>44.523743</td>\n",
       "      <td>44.523743</td>\n",
       "      <td>39.189999</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>AXP</td>\n",
       "      <td>18.570000</td>\n",
       "      <td>19.520000</td>\n",
       "      <td>18.400000</td>\n",
       "      <td>15.477420</td>\n",
       "      <td>10955700.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.990894</td>\n",
       "      <td>2.660558</td>\n",
       "      <td>100.0</td>\n",
       "      <td>66.666667</td>\n",
       "      <td>100.0</td>\n",
       "      <td>15.477420</td>\n",
       "      <td>15.477420</td>\n",
       "      <td>39.189999</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>BA</td>\n",
       "      <td>42.799999</td>\n",
       "      <td>45.560001</td>\n",
       "      <td>42.779999</td>\n",
       "      <td>33.941086</td>\n",
       "      <td>7010200.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.990894</td>\n",
       "      <td>2.660558</td>\n",
       "      <td>100.0</td>\n",
       "      <td>66.666667</td>\n",
       "      <td>100.0</td>\n",
       "      <td>33.941086</td>\n",
       "      <td>33.941086</td>\n",
       "      <td>39.189999</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>CAT</td>\n",
       "      <td>44.910000</td>\n",
       "      <td>46.980000</td>\n",
       "      <td>44.709999</td>\n",
       "      <td>31.942247</td>\n",
       "      <td>7117200.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.990894</td>\n",
       "      <td>2.660558</td>\n",
       "      <td>100.0</td>\n",
       "      <td>66.666667</td>\n",
       "      <td>100.0</td>\n",
       "      <td>31.942247</td>\n",
       "      <td>31.942247</td>\n",
       "      <td>39.189999</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>CRM</td>\n",
       "      <td>8.025000</td>\n",
       "      <td>8.550000</td>\n",
       "      <td>7.912500</td>\n",
       "      <td>8.505000</td>\n",
       "      <td>4069200.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.990894</td>\n",
       "      <td>2.660558</td>\n",
       "      <td>100.0</td>\n",
       "      <td>66.666667</td>\n",
       "      <td>100.0</td>\n",
       "      <td>8.505000</td>\n",
       "      <td>8.505000</td>\n",
       "      <td>39.189999</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>CSCO</td>\n",
       "      <td>16.410000</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>16.250000</td>\n",
       "      <td>12.155674</td>\n",
       "      <td>40980600.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.990894</td>\n",
       "      <td>2.660558</td>\n",
       "      <td>100.0</td>\n",
       "      <td>66.666667</td>\n",
       "      <td>100.0</td>\n",
       "      <td>12.155674</td>\n",
       "      <td>12.155674</td>\n",
       "      <td>39.189999</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>CVX</td>\n",
       "      <td>74.230003</td>\n",
       "      <td>77.300003</td>\n",
       "      <td>73.580002</td>\n",
       "      <td>44.404152</td>\n",
       "      <td>13695900.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.990894</td>\n",
       "      <td>2.660558</td>\n",
       "      <td>100.0</td>\n",
       "      <td>66.666667</td>\n",
       "      <td>100.0</td>\n",
       "      <td>44.404152</td>\n",
       "      <td>44.404152</td>\n",
       "      <td>39.189999</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>DIS</td>\n",
       "      <td>22.760000</td>\n",
       "      <td>24.030001</td>\n",
       "      <td>22.500000</td>\n",
       "      <td>20.597498</td>\n",
       "      <td>9796600.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.990894</td>\n",
       "      <td>2.660558</td>\n",
       "      <td>100.0</td>\n",
       "      <td>66.666667</td>\n",
       "      <td>100.0</td>\n",
       "      <td>20.597498</td>\n",
       "      <td>20.597498</td>\n",
       "      <td>39.189999</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2009-01-02</td>\n",
       "      <td>GS</td>\n",
       "      <td>84.019997</td>\n",
       "      <td>87.620003</td>\n",
       "      <td>82.190002</td>\n",
       "      <td>70.205009</td>\n",
       "      <td>14088500.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.990894</td>\n",
       "      <td>2.660558</td>\n",
       "      <td>100.0</td>\n",
       "      <td>66.666667</td>\n",
       "      <td>100.0</td>\n",
       "      <td>70.205009</td>\n",
       "      <td>70.205009</td>\n",
       "      <td>39.189999</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cb68e2b7-fcdf-4531-bf90-cce0224fb057')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-cb68e2b7-fcdf-4531-bf90-cce0224fb057 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-cb68e2b7-fcdf-4531-bf90-cce0224fb057');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "         date   tic       open       high        low      close       volume  \\\n",
       "0  2009-01-02  AAPL   3.067143   3.251429   3.041429   2.767330  746015200.0   \n",
       "1  2009-01-02  AMGN  58.590000  59.080002  57.750000  44.523743    6547900.0   \n",
       "2  2009-01-02   AXP  18.570000  19.520000  18.400000  15.477420   10955700.0   \n",
       "3  2009-01-02    BA  42.799999  45.560001  42.779999  33.941086    7010200.0   \n",
       "4  2009-01-02   CAT  44.910000  46.980000  44.709999  31.942247    7117200.0   \n",
       "5  2009-01-02   CRM   8.025000   8.550000   7.912500   8.505000    4069200.0   \n",
       "6  2009-01-02  CSCO  16.410000  17.000000  16.250000  12.155674   40980600.0   \n",
       "7  2009-01-02   CVX  74.230003  77.300003  73.580002  44.404152   13695900.0   \n",
       "8  2009-01-02   DIS  22.760000  24.030001  22.500000  20.597498    9796600.0   \n",
       "9  2009-01-02    GS  84.019997  87.620003  82.190002  70.205009   14088500.0   \n",
       "\n",
       "   day  macd   boll_ub   boll_lb  rsi_30     cci_30  dx_30  close_30_sma  \\\n",
       "0  4.0   0.0  2.990894  2.660558   100.0  66.666667  100.0      2.767330   \n",
       "1  4.0   0.0  2.990894  2.660558   100.0  66.666667  100.0     44.523743   \n",
       "2  4.0   0.0  2.990894  2.660558   100.0  66.666667  100.0     15.477420   \n",
       "3  4.0   0.0  2.990894  2.660558   100.0  66.666667  100.0     33.941086   \n",
       "4  4.0   0.0  2.990894  2.660558   100.0  66.666667  100.0     31.942247   \n",
       "5  4.0   0.0  2.990894  2.660558   100.0  66.666667  100.0      8.505000   \n",
       "6  4.0   0.0  2.990894  2.660558   100.0  66.666667  100.0     12.155674   \n",
       "7  4.0   0.0  2.990894  2.660558   100.0  66.666667  100.0     44.404152   \n",
       "8  4.0   0.0  2.990894  2.660558   100.0  66.666667  100.0     20.597498   \n",
       "9  4.0   0.0  2.990894  2.660558   100.0  66.666667  100.0     70.205009   \n",
       "\n",
       "   close_60_sma        vix  turbulence  \n",
       "0      2.767330  39.189999         0.0  \n",
       "1     44.523743  39.189999         0.0  \n",
       "2     15.477420  39.189999         0.0  \n",
       "3     33.941086  39.189999         0.0  \n",
       "4     31.942247  39.189999         0.0  \n",
       "5      8.505000  39.189999         0.0  \n",
       "6     12.155674  39.189999         0.0  \n",
       "7     44.404152  39.189999         0.0  \n",
       "8     20.597498  39.189999         0.0  \n",
       "9     70.205009  39.189999         0.0  "
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "processed_full.sort_values(['date','tic'],ignore_index=True).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-QsYaY0Dh1iw"
   },
   "source": [
    "<a id='4'></a>\n",
    "# Part 5. Build A Market Environment in OpenAI Gym-style\n",
    "The training process involves observing stock price change, taking an action and reward's calculation. By interacting with the market environment, the agent will eventually derive a trading strategy that may maximize (expected) rewards.\n",
    "\n",
    "Our market environment, based on OpenAI Gym, simulates stock markets with historical market data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5TOhcryx44bb"
   },
   "source": [
    "## Data Split\n",
    "We split the data into training set and testing set as follows:\n",
    "\n",
    "Training data period: 2009-01-01 to 2020-07-01\n",
    "\n",
    "Trading data period: 2020-07-01 to 2021-10-31\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "W0qaVGjLtgbI",
    "outputId": "891ef360-4e53-4e9c-e7f6-2bf314487d20"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "83897\n",
      "9744\n"
     ]
    }
   ],
   "source": [
    "train = data_split(processed_full, TRAIN_START_DATE,TRAIN_END_DATE)\n",
    "trade = data_split(processed_full, TRADE_START_DATE,TRADE_END_DATE)\n",
    "print(len(train))\n",
    "print(len(trade))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "id": "p52zNCOhTtLR",
    "outputId": "13221d6b-b642-4fa7-ff13-112ec9fc28c8"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-43e18d46-c070-4920-9e5e-ecdf63d7f3a3\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>tic</th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>day</th>\n",
       "      <th>macd</th>\n",
       "      <th>boll_ub</th>\n",
       "      <th>boll_lb</th>\n",
       "      <th>rsi_30</th>\n",
       "      <th>cci_30</th>\n",
       "      <th>dx_30</th>\n",
       "      <th>close_30_sma</th>\n",
       "      <th>close_60_sma</th>\n",
       "      <th>vix</th>\n",
       "      <th>turbulence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2892</th>\n",
       "      <td>2020-06-30</td>\n",
       "      <td>UNH</td>\n",
       "      <td>288.570007</td>\n",
       "      <td>296.450012</td>\n",
       "      <td>287.660004</td>\n",
       "      <td>286.754150</td>\n",
       "      <td>2932900.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.019409</td>\n",
       "      <td>302.845880</td>\n",
       "      <td>270.287341</td>\n",
       "      <td>52.413038</td>\n",
       "      <td>-25.866360</td>\n",
       "      <td>1.846804</td>\n",
       "      <td>286.997196</td>\n",
       "      <td>280.002894</td>\n",
       "      <td>30.43</td>\n",
       "      <td>12.918752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2892</th>\n",
       "      <td>2020-06-30</td>\n",
       "      <td>V</td>\n",
       "      <td>191.490005</td>\n",
       "      <td>193.750000</td>\n",
       "      <td>190.160004</td>\n",
       "      <td>190.399994</td>\n",
       "      <td>9040100.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.046930</td>\n",
       "      <td>198.399130</td>\n",
       "      <td>184.714223</td>\n",
       "      <td>53.021029</td>\n",
       "      <td>-51.567948</td>\n",
       "      <td>2.013358</td>\n",
       "      <td>191.146478</td>\n",
       "      <td>181.356465</td>\n",
       "      <td>30.43</td>\n",
       "      <td>12.918752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2892</th>\n",
       "      <td>2020-06-30</td>\n",
       "      <td>VZ</td>\n",
       "      <td>54.919998</td>\n",
       "      <td>55.290001</td>\n",
       "      <td>54.360001</td>\n",
       "      <td>49.750816</td>\n",
       "      <td>17414800.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.431681</td>\n",
       "      <td>53.248505</td>\n",
       "      <td>48.123870</td>\n",
       "      <td>48.097019</td>\n",
       "      <td>-51.065667</td>\n",
       "      <td>8.508886</td>\n",
       "      <td>50.378310</td>\n",
       "      <td>50.825243</td>\n",
       "      <td>30.43</td>\n",
       "      <td>12.918752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2892</th>\n",
       "      <td>2020-06-30</td>\n",
       "      <td>WBA</td>\n",
       "      <td>42.119999</td>\n",
       "      <td>42.580002</td>\n",
       "      <td>41.759998</td>\n",
       "      <td>38.577282</td>\n",
       "      <td>4782100.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.082999</td>\n",
       "      <td>42.108878</td>\n",
       "      <td>36.058569</td>\n",
       "      <td>48.830196</td>\n",
       "      <td>-14.542000</td>\n",
       "      <td>1.500723</td>\n",
       "      <td>38.675563</td>\n",
       "      <td>38.477853</td>\n",
       "      <td>30.43</td>\n",
       "      <td>12.918752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2892</th>\n",
       "      <td>2020-06-30</td>\n",
       "      <td>WMT</td>\n",
       "      <td>119.220001</td>\n",
       "      <td>120.129997</td>\n",
       "      <td>118.540001</td>\n",
       "      <td>115.618225</td>\n",
       "      <td>6836400.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.882725</td>\n",
       "      <td>118.955683</td>\n",
       "      <td>113.018221</td>\n",
       "      <td>48.159678</td>\n",
       "      <td>-69.952642</td>\n",
       "      <td>3.847271</td>\n",
       "      <td>117.276855</td>\n",
       "      <td>119.204110</td>\n",
       "      <td>30.43</td>\n",
       "      <td>12.918752</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-43e18d46-c070-4920-9e5e-ecdf63d7f3a3')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-43e18d46-c070-4920-9e5e-ecdf63d7f3a3 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-43e18d46-c070-4920-9e5e-ecdf63d7f3a3');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "            date  tic        open        high         low       close  \\\n",
       "2892  2020-06-30  UNH  288.570007  296.450012  287.660004  286.754150   \n",
       "2892  2020-06-30    V  191.490005  193.750000  190.160004  190.399994   \n",
       "2892  2020-06-30   VZ   54.919998   55.290001   54.360001   49.750816   \n",
       "2892  2020-06-30  WBA   42.119999   42.580002   41.759998   38.577282   \n",
       "2892  2020-06-30  WMT  119.220001  120.129997  118.540001  115.618225   \n",
       "\n",
       "          volume  day      macd     boll_ub     boll_lb     rsi_30     cci_30  \\\n",
       "2892   2932900.0  1.0 -0.019409  302.845880  270.287341  52.413038 -25.866360   \n",
       "2892   9040100.0  1.0  1.046930  198.399130  184.714223  53.021029 -51.567948   \n",
       "2892  17414800.0  1.0 -0.431681   53.248505   48.123870  48.097019 -51.065667   \n",
       "2892   4782100.0  1.0 -0.082999   42.108878   36.058569  48.830196 -14.542000   \n",
       "2892   6836400.0  1.0 -0.882725  118.955683  113.018221  48.159678 -69.952642   \n",
       "\n",
       "         dx_30  close_30_sma  close_60_sma    vix  turbulence  \n",
       "2892  1.846804    286.997196    280.002894  30.43   12.918752  \n",
       "2892  2.013358    191.146478    181.356465  30.43   12.918752  \n",
       "2892  8.508886     50.378310     50.825243  30.43   12.918752  \n",
       "2892  1.500723     38.675563     38.477853  30.43   12.918752  \n",
       "2892  3.847271    117.276855    119.204110  30.43   12.918752  "
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "id": "k9zU9YaTTvFq",
    "outputId": "4cdb6ae2-4a6e-4ea4-92f3-619bd53f4f90"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-f76bd4b5-2d5d-43fa-ae4a-9bd85bcb0d47\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>tic</th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>day</th>\n",
       "      <th>macd</th>\n",
       "      <th>boll_ub</th>\n",
       "      <th>boll_lb</th>\n",
       "      <th>rsi_30</th>\n",
       "      <th>cci_30</th>\n",
       "      <th>dx_30</th>\n",
       "      <th>close_30_sma</th>\n",
       "      <th>close_60_sma</th>\n",
       "      <th>vix</th>\n",
       "      <th>turbulence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-07-01</td>\n",
       "      <td>AAPL</td>\n",
       "      <td>91.279999</td>\n",
       "      <td>91.839996</td>\n",
       "      <td>90.977501</td>\n",
       "      <td>89.779900</td>\n",
       "      <td>110737200.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.010425</td>\n",
       "      <td>92.570741</td>\n",
       "      <td>80.068726</td>\n",
       "      <td>62.807156</td>\n",
       "      <td>107.494072</td>\n",
       "      <td>29.730532</td>\n",
       "      <td>83.817340</td>\n",
       "      <td>77.609739</td>\n",
       "      <td>28.620001</td>\n",
       "      <td>53.068171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-07-01</td>\n",
       "      <td>AMGN</td>\n",
       "      <td>235.520004</td>\n",
       "      <td>256.230011</td>\n",
       "      <td>232.580002</td>\n",
       "      <td>238.313553</td>\n",
       "      <td>6575800.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.608527</td>\n",
       "      <td>230.616459</td>\n",
       "      <td>198.678637</td>\n",
       "      <td>61.279654</td>\n",
       "      <td>271.208651</td>\n",
       "      <td>46.806139</td>\n",
       "      <td>213.212108</td>\n",
       "      <td>214.276888</td>\n",
       "      <td>28.620001</td>\n",
       "      <td>53.068171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-07-01</td>\n",
       "      <td>AXP</td>\n",
       "      <td>95.250000</td>\n",
       "      <td>96.959999</td>\n",
       "      <td>93.639999</td>\n",
       "      <td>91.743050</td>\n",
       "      <td>3301000.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>-0.387711</td>\n",
       "      <td>110.012240</td>\n",
       "      <td>87.432132</td>\n",
       "      <td>48.504820</td>\n",
       "      <td>-66.321030</td>\n",
       "      <td>3.142448</td>\n",
       "      <td>96.882072</td>\n",
       "      <td>90.357378</td>\n",
       "      <td>28.620001</td>\n",
       "      <td>53.068171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-07-01</td>\n",
       "      <td>BA</td>\n",
       "      <td>185.880005</td>\n",
       "      <td>190.610001</td>\n",
       "      <td>180.039993</td>\n",
       "      <td>180.320007</td>\n",
       "      <td>49036700.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.443193</td>\n",
       "      <td>220.721139</td>\n",
       "      <td>160.932863</td>\n",
       "      <td>50.925771</td>\n",
       "      <td>24.220608</td>\n",
       "      <td>15.932920</td>\n",
       "      <td>176.472335</td>\n",
       "      <td>155.614168</td>\n",
       "      <td>28.620001</td>\n",
       "      <td>53.068171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-07-01</td>\n",
       "      <td>CAT</td>\n",
       "      <td>129.380005</td>\n",
       "      <td>129.399994</td>\n",
       "      <td>125.879997</td>\n",
       "      <td>119.817116</td>\n",
       "      <td>2807800.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.263828</td>\n",
       "      <td>129.720775</td>\n",
       "      <td>112.569077</td>\n",
       "      <td>52.865423</td>\n",
       "      <td>35.633567</td>\n",
       "      <td>14.457404</td>\n",
       "      <td>118.586875</td>\n",
       "      <td>112.860600</td>\n",
       "      <td>28.620001</td>\n",
       "      <td>53.068171</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f76bd4b5-2d5d-43fa-ae4a-9bd85bcb0d47')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-f76bd4b5-2d5d-43fa-ae4a-9bd85bcb0d47 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-f76bd4b5-2d5d-43fa-ae4a-9bd85bcb0d47');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "         date   tic        open        high         low       close  \\\n",
       "0  2020-07-01  AAPL   91.279999   91.839996   90.977501   89.779900   \n",
       "0  2020-07-01  AMGN  235.520004  256.230011  232.580002  238.313553   \n",
       "0  2020-07-01   AXP   95.250000   96.959999   93.639999   91.743050   \n",
       "0  2020-07-01    BA  185.880005  190.610001  180.039993  180.320007   \n",
       "0  2020-07-01   CAT  129.380005  129.399994  125.879997  119.817116   \n",
       "\n",
       "        volume  day      macd     boll_ub     boll_lb     rsi_30      cci_30  \\\n",
       "0  110737200.0  2.0  3.010425   92.570741   80.068726  62.807156  107.494072   \n",
       "0    6575800.0  2.0  3.608527  230.616459  198.678637  61.279654  271.208651   \n",
       "0    3301000.0  2.0 -0.387711  110.012240   87.432132  48.504820  -66.321030   \n",
       "0   49036700.0  2.0  5.443193  220.721139  160.932863  50.925771   24.220608   \n",
       "0    2807800.0  2.0  1.263828  129.720775  112.569077  52.865423   35.633567   \n",
       "\n",
       "       dx_30  close_30_sma  close_60_sma        vix  turbulence  \n",
       "0  29.730532     83.817340     77.609739  28.620001   53.068171  \n",
       "0  46.806139    213.212108    214.276888  28.620001   53.068171  \n",
       "0   3.142448     96.882072     90.357378  28.620001   53.068171  \n",
       "0  15.932920    176.472335    155.614168  28.620001   53.068171  \n",
       "0  14.457404    118.586875    112.860600  28.620001   53.068171  "
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trade.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zYN573SOHhxG",
    "outputId": "ad22aad1-6e8b-4ae0-d053-27177e8464fc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['macd',\n",
       " 'boll_ub',\n",
       " 'boll_lb',\n",
       " 'rsi_30',\n",
       " 'cci_30',\n",
       " 'dx_30',\n",
       " 'close_30_sma',\n",
       " 'close_60_sma']"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "INDICATORS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Q2zqII8rMIqn",
    "outputId": "05d16de1-86cf-470e-b26c-df2087ffa1be"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stock Dimension: 29, State Space: 291\n"
     ]
    }
   ],
   "source": [
    "stock_dimension = len(train.tic.unique())\n",
    "state_space = 1 + 2*stock_dimension + len(INDICATORS)*stock_dimension\n",
    "print(f\"Stock Dimension: {stock_dimension}, State Space: {state_space}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AWyp84Ltto19"
   },
   "outputs": [],
   "source": [
    "buy_cost_list = sell_cost_list = [0.001] * stock_dimension\n",
    "num_stock_shares = [0] * stock_dimension\n",
    "\n",
    "env_kwargs = {\n",
    "    \"hmax\": 100,\n",
    "    \"initial_amount\": 1000000,\n",
    "    \"num_stock_shares\": num_stock_shares,\n",
    "    \"buy_cost_pct\": buy_cost_list,\n",
    "    \"sell_cost_pct\": sell_cost_list,\n",
    "    \"state_space\": state_space,\n",
    "    \"stock_dim\": stock_dimension,\n",
    "    \"tech_indicator_list\": INDICATORS,\n",
    "    \"action_space\": stock_dimension,\n",
    "    \"reward_scaling\": 1e-4\n",
    "}\n",
    "\n",
    "\n",
    "e_train_gym = StockTradingEnv(df = train, **env_kwargs)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "64EoqOrQjiVf"
   },
   "source": [
    "## Environment for Training\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xwSvvPjutpqS",
    "outputId": "89fa5b57-123e-408f-8d30-bd6cc583caa9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'stable_baselines3.common.vec_env.dummy_vec_env.DummyVecEnv'>\n"
     ]
    }
   ],
   "source": [
    "env_train, _ = e_train_gym.get_sb_env()\n",
    "print(type(env_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HMNR5nHjh1iz"
   },
   "source": [
    "<a id='5'></a>\n",
    "# Part 6: Train DRL Agents\n",
    "* The DRL algorithms are from **Stable Baselines 3**. Users are also encouraged to try **ElegantRL** and **Ray RLlib**.\n",
    "* FinRL includes fine-tuned standard DRL algorithms, such as DQN, DDPG, Multi-Agent DDPG, PPO, SAC, A2C and TD3. We also allow users to\n",
    "design their own DRL algorithms by adapting these DRL algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 240
    },
    "id": "364PsqckttcQ",
    "outputId": "1725c536-2ffe-420b-e70d-2043fb83c3b2"
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-86-a5629c21eead>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0magent\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mDRLAgent\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0menv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mif_using_ddpg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mif_using_ppo\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mif_using_td3\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: __init__() missing 3 required positional arguments: 'price_array', 'tech_array', and 'turbulence_array'"
     ]
    }
   ],
   "source": [
    "agent = DRLAgent(env = env_train, price_array=)\n",
    "\n",
    "if_using_a2c = False\n",
    "if_using_ddpg = False\n",
    "if_using_ppo = False\n",
    "if_using_td3 = False\n",
    "if_using_sac = True\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YDmqOyF9h1iz"
   },
   "source": [
    "### Agent Training: 5 algorithms (A2C, DDPG, PPO, TD3, SAC)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uijiWgkuh1jB"
   },
   "source": [
    "### Agent 1: A2C\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GUCnkn-HIbmj"
   },
   "outputs": [],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "model_a2c = agent.get_model(\"a2c\")\n",
    "\n",
    "if if_using_a2c:\n",
    "  # set up logger\n",
    "  tmp_path = RESULTS_DIR + '/a2c'\n",
    "  new_logger_a2c = configure(tmp_path, [\"stdout\", \"csv\", \"tensorboard\"])\n",
    "  # Set new logger\n",
    "  model_a2c.set_logger(new_logger_a2c)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0GVpkWGqH4-D"
   },
   "outputs": [],
   "source": [
    "trained_a2c = agent.train_model(model=model_a2c, \n",
    "                             tb_log_name='a2c',\n",
    "                             total_timesteps=50000) if if_using_a2c else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MRiOtrywfAo1"
   },
   "source": [
    "### Agent 2: DDPG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "M2YadjfnLwgt"
   },
   "outputs": [],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "model_ddpg = agent.get_model(\"ddpg\")\n",
    "\n",
    "if if_using_ddpg:\n",
    "  # set up logger\n",
    "  tmp_path = RESULTS_DIR + '/ddpg'\n",
    "  new_logger_ddpg = configure(tmp_path, [\"stdout\", \"csv\", \"tensorboard\"])\n",
    "  # Set new logger\n",
    "  model_ddpg.set_logger(new_logger_ddpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tCDa78rqfO_a"
   },
   "outputs": [],
   "source": [
    "trained_ddpg = agent.train_model(model=model_ddpg, \n",
    "                             tb_log_name='ddpg',\n",
    "                             total_timesteps=50000) if if_using_ddpg else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_gDkU-j-fCmZ"
   },
   "source": [
    "### Agent 3: PPO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "y5D5PFUhMzSV"
   },
   "outputs": [],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "PPO_PARAMS = {\n",
    "    \"n_steps\": 2048,\n",
    "    \"ent_coef\": 0.01,\n",
    "    \"learning_rate\": 0.00025,\n",
    "    \"batch_size\": 128,\n",
    "}\n",
    "model_ppo = agent.get_model(\"ppo\",model_kwargs = PPO_PARAMS)\n",
    "\n",
    "if if_using_ppo:\n",
    "  # set up logger\n",
    "  tmp_path = RESULTS_DIR + '/a2c'\n",
    "  new_logger_ppo = configure(tmp_path, [\"stdout\", \"csv\", \"tensorboard\"])\n",
    "  # Set new logger\n",
    "  model_ppo.set_logger(new_logger_ppo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Gt8eIQKYM4G3"
   },
   "outputs": [],
   "source": [
    "trained_ppo = agent.train_model(model=model_ppo, \n",
    "                             tb_log_name='ppo',\n",
    "                             total_timesteps=50000) if if_using_ppo else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3Zpv4S0-fDBv"
   },
   "source": [
    "### Agent 4: TD3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JSAHhV4Xc-bh"
   },
   "outputs": [],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "TD3_PARAMS = {\"batch_size\": 100, \n",
    "              \"buffer_size\": 1000000, \n",
    "              \"learning_rate\": 0.001}\n",
    "\n",
    "model_td3 = agent.get_model(\"td3\",model_kwargs = TD3_PARAMS)\n",
    "\n",
    "if if_using_td3:\n",
    "  # set up logger\n",
    "  tmp_path = RESULTS_DIR + '/td3'\n",
    "  new_logger_td3 = configure(tmp_path, [\"stdout\", \"csv\", \"tensorboard\"])\n",
    "  # Set new logger\n",
    "  model_td3.set_logger(new_logger_td3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OSRxNYAxdKpU"
   },
   "outputs": [],
   "source": [
    "trained_td3 = agent.train_model(model=model_td3, \n",
    "                             tb_log_name='td3',\n",
    "                             total_timesteps=30000) if if_using_td3 else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Dr49PotrfG01"
   },
   "source": [
    "### Agent 5: SAC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xwOhVjqRkCdM"
   },
   "outputs": [],
   "source": [
    "agent = DRLAgent(env = env_train)\n",
    "SAC_PARAMS = {\n",
    "    \"batch_size\": 128,\n",
    "    \"buffer_size\": 100000,\n",
    "    \"learning_rate\": 0.0001,\n",
    "    \"learning_starts\": 100,\n",
    "    \"ent_coef\": \"auto_0.1\",\n",
    "}\n",
    "\n",
    "model_sac = agent.get_model(\"sac\",model_kwargs = SAC_PARAMS)\n",
    "\n",
    "if if_using_sac:\n",
    "  # set up logger\n",
    "  tmp_path = RESULTS_DIR + '/sac'\n",
    "  new_logger_sac = configure(tmp_path, [\"stdout\", \"csv\", \"tensorboard\"])\n",
    "  # Set new logger\n",
    "  model_sac.set_logger(new_logger_sac)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "K8RSdKCckJyH"
   },
   "outputs": [],
   "source": [
    "trained_sac = agent.train_model(model=model_sac, \n",
    "                             tb_log_name='sac',\n",
    "                             total_timesteps=40000) if if_using_sac else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f2wZgkQXh1jE"
   },
   "source": [
    "## In-sample Performance\n",
    "\n",
    "Assume that the initial capital is $1,000,000."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bEv5KGC8h1jE"
   },
   "source": [
    "### Set turbulence threshold\n",
    "Set the turbulence threshold to be greater than the maximum of insample turbulence data. If current turbulence index is greater than the threshold, then we assume that the current market is volatile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "efwBi84ch1jE"
   },
   "outputs": [],
   "source": [
    "data_risk_indicator = processed_full[(processed_full.date<TRAIN_END_DATE) & (processed_full.date>=TRAIN_START_DATE)]\n",
    "insample_risk_indicator = data_risk_indicator.drop_duplicates(subset=['date'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VHZMBpSqh1jG"
   },
   "outputs": [],
   "source": [
    "insample_risk_indicator.vix.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BDkszkMloRWT"
   },
   "outputs": [],
   "source": [
    "insample_risk_indicator.vix.quantile(0.996)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AL7hs7svnNWT"
   },
   "outputs": [],
   "source": [
    "insample_risk_indicator.turbulence.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "N78hfHckoqJ9"
   },
   "outputs": [],
   "source": [
    "insample_risk_indicator.turbulence.quantile(0.996)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U5mmgQF_h1jQ"
   },
   "source": [
    "### Trading (Out-of-sample Performance)\n",
    "\n",
    "We update periodically in order to take full advantage of the data, e.g., retrain quarterly, monthly or weekly. We also tune the parameters along the way, in this notebook we use the in-sample data from 2009-01 to 2020-07 to tune the parameters once, so there is some alpha decay here as the length of trade date extends. \n",
    "\n",
    "Numerous hyperparameters – e.g. the learning rate, the total number of samples to train on – influence the learning process and are usually determined by testing some variations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cIqoV0GSI52v"
   },
   "outputs": [],
   "source": [
    "e_trade_gym = StockTradingEnv(df = trade, turbulence_threshold = 70,risk_indicator_col='vix', **env_kwargs)\n",
    "# env_trade, obs_trade = e_trade_gym.get_sb_env()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "W_XNgGsBMeVw"
   },
   "outputs": [],
   "source": [
    "trade.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "eLOnL5eYh1jR"
   },
   "outputs": [],
   "source": [
    "trained_moedl = trained_sac\n",
    "df_account_value, df_actions = DRLAgent.DRL_prediction(\n",
    "    model=trained_moedl, \n",
    "    environment = e_trade_gym)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ERxw3KqLkcP4"
   },
   "outputs": [],
   "source": [
    "df_account_value.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2yRkNguY5yvp"
   },
   "outputs": [],
   "source": [
    "df_account_value.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nFlK5hNbWVFk"
   },
   "outputs": [],
   "source": [
    "df_actions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W6vvNSC6h1jZ"
   },
   "source": [
    "<a id='6'></a>\n",
    "# Part 7: Backtesting Results\n",
    "Backtesting plays a key role in evaluating the performance of a trading strategy. Automated backtesting tool is preferred because it reduces the human error. We usually use the Quantopian pyfolio package to backtest our trading strategies. It is easy to use and consists of various individual plots that provide a comprehensive image of the performance of a trading strategy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Lr2zX7ZxNyFQ"
   },
   "source": [
    "<a id='6.1'></a>\n",
    "## 7.1 BackTestStats\n",
    "pass in df_account_value, this information is stored in env class\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Nzkr9yv-AdV_"
   },
   "outputs": [],
   "source": [
    "print(\"==============Get Backtest Results===========\")\n",
    "now = datetime.datetime.now().strftime('%Y%m%d-%Hh%M')\n",
    "\n",
    "perf_stats_all = backtest_stats(account_value=df_account_value)\n",
    "perf_stats_all = pd.DataFrame(perf_stats_all)\n",
    "perf_stats_all.to_csv(\"./\"+RESULTS_DIR+\"/perf_stats_all_\"+now+'.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QkV-LB66iwhD"
   },
   "outputs": [],
   "source": [
    "#baseline stats\n",
    "print(\"==============Get Baseline Stats===========\")\n",
    "baseline_df = get_baseline(\n",
    "        ticker=\"^DJI\", \n",
    "        start = df_account_value.loc[0,'date'],\n",
    "        end = df_account_value.loc[len(df_account_value)-1,'date'])\n",
    "\n",
    "stats = backtest_stats(baseline_df, value_col_name = 'close')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qg1kvfemrrQH"
   },
   "outputs": [],
   "source": [
    "df_account_value.loc[0,'date']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tt1bzL5OrsTa"
   },
   "outputs": [],
   "source": [
    "df_account_value.loc[len(df_account_value)-1,'date']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9U6Suru3h1jc"
   },
   "source": [
    "<a id='6.2'></a>\n",
    "## 7.2 BackTestPlot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lKRGftSS7pNM"
   },
   "outputs": [],
   "source": [
    "print(\"==============Compare to DJIA===========\")\n",
    "%matplotlib inline\n",
    "# S&P 500: ^GSPC\n",
    "# Dow Jones Index: ^DJI\n",
    "# NASDAQ 100: ^NDX\n",
    "backtest_plot(df_account_value, \n",
    "             baseline_ticker = '^DJI', \n",
    "             baseline_start = df_account_value.loc[0,'date'],\n",
    "             baseline_end = df_account_value.loc[len(df_account_value)-1,'date'])"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "_gDkU-j-fCmZ",
    "3Zpv4S0-fDBv"
   ],
   "name": "Stock_NeurIPS2018.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
